Gaze-based non-regular subsampling of sensor pixels

ABSTRACT

An imaging system including: image sensor including pixels arranged on photo-sensitive surface; and processor configured to: obtain information indicative of gaze direction of user&#39;s eye; identify gaze position on photo-sensitive surface; determine first region and second region on photo-sensitive surface, wherein first region includes and surrounds gaze position, while second region surrounds first region; read out first pixel data from each pixel of first region; select set of pixels to be read out from second region based on predetermined sub-sampling pattern; read out second pixel data from pixels of selected set; generate, from second pixel data, pixel data of remaining pixels of second region; and process first pixel data, second pixel data, and generated pixel data to generate image frame(s).

TECHNICAL FIELD

The present disclosure relates to imaging systems incorporatinggaze-based non-regular subsampling of sensor pixels. The presentdisclosure also relates to display apparatuses incorporating gaze-basednon-regular subsampling of sensor pixels. The present disclosure alsorelated to methods for gaze-based non-regular subsampling of sensorpixels.

BACKGROUND

Nowadays, with an increase in the number of images being captured everyday, there is an increased demand for developments in image processing.For example, such a demand may be quite high and critical in case ofevolving technologies such as immersive extended-reality (XR)technologies which are being employed in various fields such asentertainment, real estate, training, medical imaging operations,simulators, navigation, and the like. Several advancements are beingmade to develop image processing technology.

However, existing image processing technology has several limitationsassociated therewith. Firstly, the existing image processing technologyprocesses image signals captured by pixels of an image sensor of acamera in a manner that such processing requires considerable processingresources, involves a long processing time, requires high computingpower, and limits a number of pixels that can be arranged on an imagesensor for full pixel readout at a given frame rate. As an example,image signals corresponding to only about 9 million pixels on the imagesensor may be processed currently (by full pixel readout) to generateimage frames at 90 frames per second (FPS). Secondly, the existing imageprocessing technology is unable to cope with visual quality requirementsthat arise, for example, due to high-resolution (such as a resolutionhigher than or equal to 60 pixels per degree), small pixel size, highfield of view (FOV), and high frame-rate requirements (such as a framerate higher than or equal to 90 FPS) in some display devices (such as XRdevices). In an example, some XR devices may employ at least two camerasper eye to obtain images having a high FOV at a high frame rate.However, in such images, high resolution is obtained only in a narrowregion, since focal lengths of optical elements of the at least twocameras are typically modified by distortion in order to obtain the highfield of view. Resultantly, the generated images lack requisite visualquality, thereby leading to a poor, non-immersive viewing experience forthe user.

Therefore, in light of the foregoing discussion, there exists a need toovercome the aforementioned drawbacks associated with existing imageprocessing technology.

SUMMARY

The present disclosure seeks to provide an imaging system incorporatinggaze-based non-regular subsampling of sensor pixels. The presentdisclosure also seeks to provide a display apparatus incorporatinggaze-based non-regular subsampling of sensor pixels. The presentdisclosure also seeks to provide a method for gaze-based non-regularsubsampling of sensor pixels. An aim of the present disclosure is toprovide a solution that overcomes at least partially the problemsencountered in prior art.

In one aspect, an embodiment of the present disclosure provides animaging system comprising:

-   -   an image sensor comprising a plurality of pixels arranged on a        photo-sensitive surface thereof; and    -   a processor configured to:        -   obtain information indicative of a gaze direction of a            user's eye;        -   identify a gaze position on the photo-sensitive surface of            the image sensor, based on the gaze direction of the user's            eye;        -   determine a first region and a second region on the            photo-sensitive surface of the image sensor based on the            gaze position, wherein the first region includes and            surrounds the gaze position, while the second region            surrounds the first region;        -   read out first pixel data from each pixel of the first            region;        -   select a set of pixels that are to be read out from the            second region based on a predetermined sub-sampling pattern,            wherein the predetermined sub-sampling pattern indicates            locations of the pixels of the set;        -   read out second pixel data from the pixels of the selected            set;        -   generate, from the second pixel data, pixel data of            remaining pixels of the second region; and        -   process the first pixel data, the second pixel data and the            generated pixel data to generate at least one image frame.

In another aspect, an embodiment of the present disclosure provides adisplay apparatus comprising:

-   -   gaze-tracking means;    -   a light source per eye;    -   an image sensor per eye comprising a plurality of pixels        arranged on a photo-sensitive surface thereof; and    -   at least one processor configured to:        -   process gaze-tracking data, collected by the gaze-tracking            means, to determine a gaze direction of a user's eye;        -   identify a gaze position on the photo-sensitive surface of            the image sensor, based on the gaze direction of the user's            eye;        -   determine a first region and a second region on the            photo-sensitive surface of the image sensor based on the            gaze position, wherein the first region includes and            surrounds the gaze position, while the second region            surrounds the first region;        -   read out first pixel data from each pixel of the first            region;        -   select a set of pixels that are to be read out from the            second region based on a predetermined sub-sampling pattern,            wherein the predetermined sub-sampling pattern indicates            locations of the pixels of the set;        -   read out second pixel data from the pixels of the selected            set;        -   generate, from the second pixel data, pixel data of            remaining pixels of the second region;        -   process the first pixel data, the second pixel data and the            generated pixel data to generate at least one image frame;            and        -   display the at least one image frame via the light source.

In yet another aspect, an embodiment of the present disclosure providesa method comprising:

-   -   obtaining information indicative of a gaze direction of a user's        eye;    -   identifying a gaze position on a photo-sensitive surface of an        image sensor, based on the gaze direction of the user's eye;    -   determining a first region and a second region on the        photo-sensitive surface of the image sensor based on the gaze        position, wherein the first region includes and surrounds the        gaze position, while the second region surrounds the first        region;    -   reading out first pixel data from each pixel of the first        region;    -   selecting a set of pixels that are to be read out from the        second region based on a predetermined sub-sampling pattern,        wherein the predetermined sub-sampling pattern indicates        locations of the pixels of the set;    -   reading out second pixel data from the pixels of the selected        set;    -   generating, from the second pixel data, pixel data of remaining        pixels of the second region; and    -   processing the first pixel data, the second pixel data and the        generated pixel data to generate at least one image frame.

Embodiments of the present disclosure substantially eliminate or atleast partially address the aforementioned problems in the prior art,and enable efficient gaze-based non-regular subsampling of sensor pixelsto generate highly immersive and realistic image frames, in a mannerthat a high frame rate is obtained, and the processor is not excessivelycomputationally overburdened.

Additional aspects, advantages, features and objects of the presentdisclosure would be made apparent from the drawings and the detaileddescription of the illustrative embodiments construed in conjunctionwith the appended claims that follow.

It will be appreciated that features of the present disclosure aresusceptible to being combined in various combinations without departingfrom the scope of the present disclosure as defined by the appendedclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The summary above, as well as the following detailed description ofillustrative embodiments, is better understood when read in conjunctionwith the appended drawings. For the purpose of illustrating the presentdisclosure, exemplary constructions of the disclosure are shown in thedrawings. However, the present disclosure is not limited to specificmethods and instrumentalities disclosed herein. Moreover, those skilledin the art will understand that the drawings are not to scale. Whereverpossible, like elements have been indicated by identical numbers.

Embodiments of the present disclosure will now be described, by way ofexample only, with reference to the following diagrams wherein:

FIG. 1 illustrates a block diagram of an architecture of an imagingsystem, in accordance with an embodiment of the present disclosure;

FIGS. 2A and 2B illustrate block diagrams of architectures of a displayapparatus, in accordance with different embodiments of the presentdisclosure;

FIG. 3A illustrates different regions on a photo-sensitive surface of animage sensor, while FIG. 3B illustrates how pixels arranged on thephoto-sensitive surface are read out, in accordance with an embodimentof the present disclosure;

FIG. 4 illustrates a manner in which a predetermined sub-samplingpattern changes across eight image frames, in accordance with anembodiment of the present disclosure;

FIG. 5 illustrates a sub-sampling pattern that is used to generate abaseline sub-sampling pattern, in accordance with an embodiment of thepresent disclosure;

FIGS. 6A, 6B, 6C, and 6D illustrate a manner in which pixel data ofremaining to pixels of a portion of a second region on a photo-sensitivesurface is generated, in accordance with an embodiment of the presentdisclosure;

FIGS. 7A and 7B illustrate a processing pipeline indicating wheregeneration of pixel data of remaining pixels of a second region isperformed, in accordance with different embodiments of the presentdisclosure; and

FIGS. 8A and 8B illustrate steps of a method, in accordance with anembodiment of the present disclosure.

In the accompanying drawings, an underlined number is employed torepresent an item over which the underlined number is positioned or anitem to which the underlined number is adjacent. A non-underlined numberrelates to an item identified by a line linking the non-underlinednumber to the item. When a number is non-underlined and accompanied byan associated arrow, the non-underlined number is used to identify ageneral item at which the arrow is pointing.

DETAILED DESCRIPTION OF EMBODIMENTS

The following detailed description illustrates embodiments of thepresent disclosure and ways in which they can be implemented. Althoughsome modes of carrying out the present disclosure have been disclosed,those skilled in the art would recognize that other embodiments forcarrying out or practising the present disclosure are also possible.

In one aspect, an embodiment of the present disclosure provides animaging system comprising:

an image sensor comprising a plurality of pixels arranged on aphoto-sensitive surface thereof; and

a processor configured to:

-   -   obtain information indicative of a gaze direction of a user's        eye;    -   identify a gaze position on the photo-sensitive surface of the        image sensor, based on the gaze direction of the user's eye;    -   determine a first region and a second region on the        photo-sensitive surface of the image sensor based on the gaze        position, wherein the first region includes and surrounds the        gaze position, while the second region surrounds the first        region;    -   read out first pixel data from each pixel of the first region;    -   select a set of pixels that are to be read out from the second        region based on a predetermined sub-sampling pattern, wherein        the predetermined sub-sampling pattern indicates locations of        the pixels of the set;    -   read out second pixel data from the pixels of the selected set;    -   generate, from the second pixel data, pixel data of remaining        pixels of the second region; and    -   process the first pixel data, the second pixel data and the        generated pixel data to generate at least one image frame.

In another aspect, an embodiment of the present disclosure provides adisplay apparatus comprising:

-   -   gaze-tracking means;    -   a light source per eye;    -   an image sensor per eye comprising a plurality of pixels        arranged on a photo-sensitive surface thereof; and    -   at least one processor configured to:        -   process gaze-tracking data, collected by the gaze-tracking            means, to determine a gaze direction of a user's eye;        -   identify a gaze position on the photo-sensitive surface of            the image sensor, based on the gaze direction of the user's            eye;        -   determine a first region and a second region on the            photo-sensitive surface of the image sensor based on the            gaze position, wherein the first region includes and            surrounds the gaze position, while the second region            surrounds the first region;        -   read out first pixel data from each pixel of the first            region;        -   select a set of pixels that are to be read out from the            second region based on a predetermined sub-sampling pattern,            wherein the predetermined sub-sampling pattern indicates            locations of the pixels of the set;        -   read out second pixel data from the pixels of the selected            set;        -   generate, from the second pixel data, pixel data of            remaining pixels of the second region;        -   process the first pixel data, the second pixel data and the            generated pixel data to generate at least one image frame;            and        -   display the at least one image frame via the light source.

In yet another aspect, an embodiment of the present disclosure providesa method comprising:

-   -   obtaining information indicative of a gaze direction of a user's        eye;    -   identifying a gaze position on a photo-sensitive surface of an        image sensor, based on the gaze direction of the user's eye;    -   determining a first region and a second region on the        photo-sensitive surface of the image sensor based on the gaze        position, wherein the first region includes and surrounds the        gaze position, while the second region surrounds the first        region;    -   reading out first pixel data from each pixel of the first        region;    -   selecting a set of pixels that are to be read out from the        second region based on a predetermined sub-sampling pattern,        wherein the predetermined sub-sampling pattern indicates        locations of the pixels of the set;    -   reading out second pixel data from the pixels of the selected        set;    -   generating, from the second pixel data, pixel data of remaining        pixels of the second region; and    -   processing the first pixel data, the second pixel data and the        generated pixel data to generate at least one image frame.

The present disclosure provides the aforementioned imaging system, theaforementioned display apparatus, and the aforementioned method. Herein,pixel data from the plurality of pixels arranged on the photo-sensitivesurface is selectively (i.e., customisably) read out, based on gazedirection of the user's eye. The pixel data from all the pixels lying inthe first region is read out to provide a high resolution in acorresponding first portion of the at least one image frame, whereas thepixel data from the pixels lying in the second region is selectivelyread out based on the predetermined sub-sampling pattern, and thenun-read pixel data is generated. This produces a relatively lowerresolution in a corresponding second portion of the at least one imageframe. This variation in resolution mimics human eye viewingcharacteristics. Different pixel data are processed in a manner that theprocessor overburdening, delays, and excessive power consumption do notoccur during said processing. In this regard, conservation andutilization of processing resources of the processor is optimized.Furthermore, the processor effectively copes with visual qualityrequirements of various display apparatuses, to generate the at leastone image frame with a requisite visual quality. The processor can bejudiciously used to also perform other processing tasks, if required. Aselective read out of the pixel data in the second region alsofacilitates in providing a high frame rate of image frames. Thisfacilitates an increase in overall efficiency of the processor, andmakes the imaging system suitable for use with demanding applications(such as extended-reality applications). Moreover, the imaging systemcan easily employ a single camera per eye to obtain high,spatially-varying image quality of the at least one image frame thatemulates image viewing quality and characteristics of human visualsystem. The method is fast, effective, reliable and can be implementedwith ease.

The imaging system comprises specialized equipment for generating the atleast one image frame which is subsequently displayed at the displayapparatus. It will be appreciated that the imaging system generates theimage frame(s) in real time or near-real time. Then, the at least oneimage frame is communicated from the imaging system to the displayapparatus. The at least one image frame is to be presented to a user ofthe display apparatus.

Throughout the present disclosure, the term “image frame” refers tovisual content, which encompasses not only colour informationrepresented in the at least one image frame, but also other attributesassociated with the at least one image frame (for example, such as depthinformation, transparency information, luminance information, and thelike). It will be appreciated that the at least one image framerepresents a real-world scene of a real-world environment. At thedisplay apparatus, the at least one image frame could be utilized togenerate a visual scene of an extended-reality (XR) environment. Theterm “extended-reality” encompasses virtual reality (VR), augmentedreality (AR), mixed reality (MR), and the like.

The imaging system is communicably coupled with the display apparatuswirelessly and/or in a wired manner. The term “display apparatus” refersto a specialized equipment that is capable of at least displaying the atleast one image frame. Optionally, the at least one processor (of thedisplay apparatus) is configured to: superimpose virtual content uponthe at least one image frame to generate at least one XR image frame;and display the at least one XR image frame via the light source.Alternatively, optionally, the processor (of the imaging system) isconfigured to superimpose the virtual content on the at least one imageframe, prior to sending it to the display apparatus. Optionally, thedisplay apparatus is implemented as a head-mounted display (HMD). Theterm “head-mounted display” refers to specialized equipment that isconfigured to present the XR environment to the user when said HMD, inoperation, is worn by the user on his/her head. The HMD is implemented,for example, as an XR headset, a pair of XR glasses, and the like, thatis operable to display the visual scene of the XR environment to theuser.

In some implementations, the imaging system is optionally integratedwith the display apparatus. In such implementations, the imaging systemis physically coupled to the display apparatus (for example, attachedvia mechanical and/or electrical connections to components of thedisplay apparatus). For example, the image sensor may be arranged on anouter surface of the display apparatus that faces the real-worldenvironment. Optionally, in such implementations, the processor of theimaging system is implemented as the at least one processor of thedisplay apparatus. Alternatively, optionally, in such implementations,the processor of the imaging system is communicably coupled to the atleast one processor of the display apparatus.

In other implementations, the imaging system is optionally implementedon a remote device that is separate from the display apparatus. In suchimplementations, the processor of the imaging system is communicablycoupled to the at least one processor of the display apparatuswirelessly and/or in a wired manner. Optionally, the imaging system ismounted on the remote device. Examples of the remote device include, butare not limited to, a drone, a vehicle, and a robot. Optionally, theremote device is physically positioned in the real-world environment,whereas the user of the display apparatus is positioned away from (forexample, at a distance from) the remote device.

In yet other implementations, the imaging system is optionally arrangedat a given location within the real-world environment. For example, theimaging system may be arranged on a support structure and may be capableof a three-dimensional (3D) rotation (and additionally, capable of atranslation motion). Herein, the support structure can be moved to anyrequired location in the real-world environment.

Throughout the present disclosure, the term “image sensor” refers to adevice that detects light from the real-world environment at itsphoto-sensitive surface, thereby enabling the plurality of pixelsarranged on the photo-sensitive surface to capture a plurality of imagesignals. The plurality of image signals are electrical signalspertaining to the real-world scene of the real-world environment. Theplurality of image signals constitute captured pixel data of theplurality of pixels. Examples of the image sensor include, but are notlimited to, a charge-coupled device (CCD) image sensor, and acomplementary metal-oxide-semiconductor (CMOS) image sensor. It will beappreciated that the plurality of pixels could be arranged in a requiredmanner (for example, such as a rectangular two-dimensional (2D) grid, apolygonal arrangement, a circular arrangement, an ellipticalarrangement, a freeform arrangement, and the like) on thephoto-sensitive surface of the image sensor. In a first example, theimage sensor may comprise 25 megapixels arranged in the rectangular 2Dgrid (such as a 5000×5000 grid) on the photo-sensitive surface.

It will be appreciated that the image sensor is a part of at least onecamera. The processor of the imaging system may also be a part of the atleast one camera. The at least one camera could be arranged anywhere inthe real-world environment where the user is present, or could bearranged on the remote device present in the real-world environment, orcould be arranged on the display apparatus worn by the user on his/herhead. Optionally, the at least one camera is implemented as at least onevisible light camera. Examples of a given visible light camera include,but are not limited to, a Red-Green-Blue (RGB) camera, aRed-Green-Blue-Alpha (RGB-A) camera, a monochrome camera, aRed-Green-Green-Blue (RGGB) camera, a Red-Yellow-Yellow-Blue (RYYB)camera, a Red-Clear-Clear-Blue (RCCB) camera, a Red-Green-Blue-Infrared(RGB-IR) camera. As an example, the RGB-IR camera can be a 2×2pattern-based RGB-IR camera, a 4×4 pattern-based RGB-IR camera, orsimilar. It will be appreciated that the at least one camera could beimplemented as a combination of the given visible light camera and adepth camera. Examples of the depth camera include, but are not limitedto, a Red-Green-Blue-Depth (RGB-D) camera, a ranging camera, a LightDetection and Ranging (LiDAR) camera, a Time-of-Flight (ToF) camera, aSound Navigation and Ranging (SONAR) camera, a laser rangefinder, astereo camera, a plenoptic camera, an infrared camera. As an example,the at least one camera may be implemented as the stereo camera. Asanother example, the at least one camera may have an image sensor having576 million pixels, wherein 2×2 grids, 3×3 grids or even 4×4 grids ofpixels of a same colour can be binned to form a single super pixel. Suchan implementation of the image sensor provides a native pixelresolution. The image sensor may, for example, have several QUAD/4Ccolour pixels in grids, wherein QUAD/4C colour pixels in each grid areto be binned to form a corresponding super pixel. Each grid of QUAD/4Ccolour pixels has four pixels of a same colour being arranged next toeach other in a 2×2 manner, wherein these four pixels are binned to forma single super pixel of the same colour.

Notably, the processor controls overall operation of the imaging system.The processor is communicably coupled to the image sensor and to thedisplay apparatus. Optionally, the processor of the imaging system isimplemented as an image signal processor. In an example, the imagesignal processor may be a programmable digital signal processor (DSP).Alternatively, optionally, the processor is implemented via a cloudserver that provides a cloud computing service.

Notably, the processor obtains, from the gaze-tracking means of thedisplay apparatus, the information indicative of the gaze direction ofthe user's eye. The term “gaze direction” refers to a direction in whichthe user's eye is gazing. The gaze direction may be represented by agaze vector. Throughout the present disclosure, the term “gaze-trackingmeans” refers to a specialized equipment for detecting and/or followinggaze of the user's eye, when the display apparatus in operation is wornby the user on his/her head. The gaze-tracking means could beimplemented as contact lenses with sensors, cameras monitoring aposition of a pupil of the user's eye, and the like. Such gaze-trackingmeans are well-known in the art. Notably, the gaze-tracking means isconfigured to collect the gaze-tracking data, which constitutes theinformation indicative of the gaze direction of the user's eye. Saidinformation may also include the gaze vector. Then, the gaze-trackingmeans optionally sends the gaze-tracking data (i.e., said information)to the processor. Optionally, the processor is configured to process thegaze-tracking data for determining the gaze direction of the user's eye.The gaze-tracking data may comprise images of the user's eye, sensorvalues, and the like. Optionally, when processing the gaze-trackingdata, the processor is configured to employ at least one of: an imageprocessing algorithm, a feature extraction algorithm, a data processingalgorithm. It will be appreciated that the gaze-tracking data iscollected repeatedly by the gaze-tracking means throughout a givensession of using the display apparatus, as gaze of the user's eye keepschanging whilst he/she uses the display apparatus. An up-to-dategaze-tracking data (indicative of the gaze direction of the user's eye)allows for generating up-to-date gaze-contingent image frame(s).

In some implementations, the gaze direction is a current gaze direction.In other implementations, the gaze direction is a predicted gazedirection. In this regard, the at least one processor of the displayapparatus predicts the gaze direction of the user's eye, and providesthe predicted gaze direction of the user's eye to the processor of theimaging system. It will be appreciated that optionally the predictedgaze direction is predicted, based on a motion of the user's gaze,wherein the predicted gaze direction lies along a direction of themotion of the user's gaze. In such a case, the motion of the user's gazecould be determined in terms of velocity and/or acceleration of theuser's gaze, using information indicative of previous gaze directions ofthe user's eye and/or the current gaze direction of the user's eye. Inyet other implementations, the gaze direction is a default gazedirection, wherein the default gaze direction of the user is straight.In such a case, it is considered that a user's gaze is, by default,typically directed towards a centre of his/her field of view. In such acase, a central portion of the user's field of view is resolved to amuch greater degree of visual detail, as compared to a peripheralportion of the user's field of view. A gaze position corresponding tothe default gaze direction lies at a centre of the photo-sensitivesurface.

Optionally, when identifying the gaze position on the photo-sensitivesurface, the processor is configured to map the gaze direction of theuser's eye onto the photo-sensitive surface. The term “gaze position”refers to a position on the photo-sensitive surface onto which the gazedirection is mapped. As an example, the gaze position may be at a centreof the photo-sensitive surface. As another example, the gaze positionmay be at a point in a top-left region of the photo-sensitive surface.

Notably, the first region and the second region are determineddynamically, based on the gaze position. In this regard, the firstregion corresponds to a gaze area (i.e., a region of interest), whereasthe second region corresponds to a peripheral area surrounding the gazearea. Such a dynamic manner of determining the first region and thesecond region emulates a way in which the user actively focuses withinhis/her field of view. It will be appreciated that some pixels fromamongst the plurality of pixels lie in the first region, while remainingpixels from amongst the plurality of pixels lie in the second region.Referring to the first example, when the gaze position is at a centre ofthe photo-sensitive surface, 1 megapixel (namely, 1 million pixels)arranged as a 1000×1000 grid in a centre may lie in the first region,while remaining 24 megapixels may lie in the second region.

Optionally, an angular extent of the first region lies in a range of 0degrees from the gaze position to 2-50 degrees from the gaze position,while an angular extent of the second region lies in a range of 12.5-50degrees from the gaze position to 45-110 degrees from the gaze position.As an example, the angular extent of the first region may be from 0degree to an angle that lies in a range of 2, 6, 10, 15, 20, 25 or 35degrees up to 10, 20, 30, 40, 45 or 50 degrees from the gaze position,while the angular extent of the second region may be from an angle thatlies in a range of 12.5, 15, 17.5, 20, 25, 30 or 35 degrees up to 20,30, 35, 40, 45 or 50 degrees from the gaze position to an angle thatlies in a range of 45, 50, 55, 60, 70, 80 or 90 degrees up to 60, 70,80, 90, 95, 100, 105 or 110 degrees from the gaze position.

Notably, the processor reads out the first pixel data from each pixel ofthe first region, since the first region corresponds to the gaze areaaccording to the gaze direction of the user's eye. When reading out thefirst pixel data, image signals captured by each pixel of first regionare processed. As a result, the first pixel data enables in achieving ahigh visual quality (i.e., a native resolution) in correspondinggaze-contingent pixels of the at least one image frame. Referring to andcontinuing from the first example, the 1 megapixel arranged as the1000×1000 grid are read out to constitute the first pixel data.Optionally, when the pixels are arranged in the rectangular 2D grid onthe photo-sensitive surface, the processor is configured to read out agiven pixel data from pixels of a given region in a line-by-line manner.

Throughout the present disclosure, the term “pixel data” refers toinformation pertaining to a given pixel of a given region, wherein saidinformation comprises one or more of: a colour value of the given pixel,a depth value of the given pixel, a transparency value of the givenpixel, a luminance value of the given pixel, and the like. Optionally,the given pixel data is in a form of a RAW image data. Alternatively,optionally, the given pixel data is in a form of a given colour spacedata. Optionally, in this regard, the processor is configured togenerate the given colour space data from the RAW image data.Optionally, the given colour space is one of: a standard Red-Green-Blue(sRGB) colour space, an RGB colour space, a Luminance and two colourdifferences (YUV) colour space, a Hue-Chroma-Luminance (HCL) colourspace, a Hue-Saturation-Lightness (HSL) colour space, aHue-Saturation-Brightness (HSB) colour space, a Hue-Saturation-Value(HSV) colour space, a Hue-Saturation-Intensity (HSI) colour space, aCyan-Magenta-Yellow-Black (CMYK) colour space, a blue-difference andred-difference chroma components (YCbCr) colour space. The RGB colourspace is optionally transformed (namely, converted) to any of theaforesaid colour spaces.

Throughout the present disclosure, the term “predetermined sub-samplingpattern” refers to a software-based masking pattern that enables inselectively reading out pixels of the second region. In this regard,pixels of the second region whose locations are indicated in thepredetermined sub-sampling pattern are selected in the set of pixelsthat are to be read out from the second region, while pixels of thesecond region whose locations are not indicated in the predeterminedsub-sampling pattern are skipped and thus, are not read out from thesecond region. The predetermined sub-sampling pattern thus provides apredetermined, non-regular selection criterion for sub-sampling pixelsof the second region. Referring to and continuing from the firstexample, the set of pixels that are to be read out from the secondregion may comprise 6 megapixels that are arranged in a non-regularmanner across the second region.

Optionally, the predetermined sub-sampling pattern is a non-regularpattern. The “non-regular pattern” is a software-based masking patternwhich indicates locations of irregularly-arranged (i.e., disorderlyarranged) pixels in the second region. These irregularly-arranged pixelsare selected as the set of pixels. Herein, the term “non-regular”indicates that pixels of the set are not selected according to anytypical or standardised spatially-regular manner, but in fact aredeliberately and carefully selected in a spatially-irregular manner soas to facilitate in accurately and reliably generating the pixel data ofthe remaining pixels of the second region. Moreover, the term“non-regular” does not imply that the pixels of the set are necessarilyselected randomly, as random selection may lead to inconsistentsub-sampling in the second region (i.e., nil or extremely lowsub-sampling in some areas of the second region and extremely highsub-sampling in some other areas of the second region).

Optionally, the non-regular pattern is generated in a manner that thenon-regular pattern is free from at least one unfavourable pixelarrangement. Optionally, in this regard, the at least one unfavourablepixel arrangement is at least one of: a 1×2 or 2×1 grid of pixels to beread out, a 2×2 grid of pixels to be read out, a 2×4 or 4×2 grid ofskipped pixels, an alternate arrangement of three pixels to be read outand two skipped pixels, a diagonal arrangement of three pixels to beread out, a zigzag arrangement of five pixels to be read out.Beneficially, when the at least one unfavourable pixel arrangement isminimized in the non-regular pattern, the pixels that are to be read outfrom the second region are optimally selected in the set. Furthermore,when the predetermined sub-sampling pattern is the non-regular pattern,undesirable visual artifacts (such as moiré effect) due to aliasing arereduced in the at least one image frame that is subsequently generated.

Optionally, the processor is configured to change the predeterminedsub-sampling pattern from one image frame to another image frame. Whenthe predetermined sub-sampling pattern is same for each image frame, theuser (of the display apparatus) viewing a sequence of image frames maybe able to perceive a same pattern of varying visual quality in thesequence of image frames, said pattern having a high-quality region(corresponding to the sampled pixels selected according to thepredetermined sub-sampling pattern) and a low-quality region(corresponding to the skipped pixels according to the predeterminedsub-sampling pattern). Therefore, the processor changes thepredetermined sub-sampling pattern from one image frame to another imageframe, such that visual quality in the sequence of image frames wouldvary differently, and such variation would be imperceptible to the user.This is because the set of pixels that are to be read out from thesecond region changes from one image frame to another image frame in aspatiotemporal manner. Furthermore, when the predetermined sub-samplingpattern is changed from one image frame to another image frame theundesirable visual artifacts due to the aliasing are considerablyreduced in sequence of image frames.

Optionally, when changing the predetermined sub-sampling pattern fromone image frame to another image frame, the processor is configured toemploy, for a given image frame, a given predetermined sub-samplingpattern that is selected from amongst a plurality of predeterminedsub-sampling patterns. Optionally, a number of the plurality ofpredetermined sub-sampling patterns is fixed, wherein the givenpredetermined sub-sampling pattern is employed in a cyclic manner fortwo or more image frames. Optionally, the number of the plurality ofpredetermined sub-sampling patterns lies in a range of 2 to 32. As anexample, the number of the plurality of predetermined sub-samplingpatterns may be from 2, 4, 9, 12, 16 or 25 up to 10, 16, 20, 25 or 32.In this regard, when a number of image frames exceeds the number of theplurality of predetermined sub-sampling patterns, the processor employsone predetermined sub-sampling pattern for one image frame until allpredetermined sub-sampling patterns are employed once, and thenre-employ the (same) plurality of predetermined sub-sampling patternsfor subsequent image frames in the cyclic manner. In an example, theprocessor may employ 8 predetermined sub-sampling patterns P1, P2, P3,P4, P5, P6, P7, and P8 for 12 image frames F1, F2, F3, F4, F5, F6, F7,F8, F9, F10, F11, and F12. Herein, the processor may employ thepredetermined sub-sampling patterns in a sequence: P1, P2, P3, P4, P5,P6, P7, P8, P1, P2, P3, P4 for the image frames F1, F1, F2, F3, F4, F5,F6, F7, F8, F9, F10, F11, F12, respectively.

Optionally, a sub-sampling density of the predetermined sub-samplingpattern varies across the second region as a function of a distance fromthe gaze position. The term “sub-sampling density” refers to a number ofpixels that are to be read out (namely, sampled) from the second regionper unit area. In this regard, said area may be expressed in terms of atotal number of pixels, a number of pixels in both horizontal andvertical dimensions, units of length, or similar. For example, thesub-sampling density may be 2 pixels per 10 pixels, 4 pixels per 4×4grid of pixels, 5 pixels per 50 square micrometres of the image sensor,or similar. Optionally, the function of the distance from the gazeposition is one of: a linear function, a non-linear function, astep-wise function. It will be appreciated that the sub-sampling densityreduces across the second region as the distance from the gaze positionincreases (i.e., the sub-sampling density across the second region ishigher near an inner periphery of the second region as compared to anouter periphery of the second region). Thus, the number of pixels thatare to be read out from the second region per unit area increase ongoing from the outer periphery of the second region towards the innerperiphery of the second region (i.e., the sub-sampling density isspatially dense near the inner periphery of the second region and isspatially sparse near the outer periphery of the second region). This isbecause the pixels of the second region lying near the gaze positionwould be perceived in the at least one image frame with high visualacuity by foveas of the user's eye, as compared to the pixels of thesecond region lying far from the gaze position. Therefore, a highersub-sampling density is required near the gaze position for accuratelyand reliably generating pixel data of the remaining pixels lying nearthe gaze position, using pixel data of the (read out) pixels lying nearthe gaze position, to produce a higher resolution.

In an example implementation, the predetermined sub-sampling patternchanges from one image frame to another image frame and, for each imageframe, the sub-sampling density of its corresponding predeterminedsub-sampling pattern varies across the second region as the function ofthe distance from the gaze position. It will also be appreciated thatwhen the sub-sampling density reduces across the second region onincrease in the distance from the gaze position, a binning ratio (i.e.,a number of pixels binned into a pixel that is sampled) and apixel-skipping ratio (i.e., a ratio of a number of skipped pixels and anumber of pixels that are read out per unit area) in the second regionalso increases as the distance from the gaze position increases. In anexample, the binning ratio near the outer periphery of the second regionmay be 16:1, 12:1, 9:1, 8:1, or similar, whereas the binning ratio nearthe inner periphery of the second region may be 6:1, 4:1, 2:1, orsimilar. The sampled pixel(s) and the pixels that are binned could bearranged as a 2×1 grid, a 2×2 grid, a 3×2 grid, a 3×3 grid, a 4×3 grid,a 4×4 grid or similar. Furthermore, a ratio of pixels of the secondregion for which pixel data is generated (i.e., hallucinated pixels) topixels of the second region that are read out increases as the distancefrom the gaze position increases.

In an embodiment, the sub-sampling density is at least 25 percent. Inthis regard, the sub-sampling density may be 25 percent, 30 percent, 35percent, 40, percent, 45 percent, 50 percent, 55 percent, 60 percent,and the like. In an example, when the sub-sampling density is at least25 percent, at least 4 pixels may be read out from amongst every 16pixels (of a 4×4 grid) of the second region. In another example, whenthe sub-sampling density is at least 50 percent, at least 2 pixels areto be read out from amongst every 4 pixels (of a 2×2 grid) of the secondregion. In an example implementation, the sub-sampling density may be 25percent near the outer periphery of the second region and may be 50percent near the inner periphery of the second region.

In another embodiment, the sub-sampling density is at most 25 percent.In this regard, the sub-sampling density may be 25 percent, 20 percent,15 percent, 10 percent, 5 percent, and the like. In an example, when thesub-sampling density is 10 percent, 10 pixels may be read out fromamongst every 100 pixels of the second region. In another example, whenthe sub-sampling density is 15 percent, only 15 pixels may be read outfrom amongst every 100 pixels of the second region. In an exampleimplementation, the sub-sampling density may be 10 percent near theouter periphery of the second region and may be 25 percent near theinner periphery of the second region.

Optionally, the processor is configured to generate the predeterminedsub-sampling pattern from a baseline sub-sampling pattern having a samesub-sampling density across the second region and indicating locationsof pixels of a baseline set, by including additional pixels in thebaseline set and indicating locations of the additional pixels, whereina number of additional pixels to be included in the baseline set perunit area increases on going from the outer periphery of the secondregion towards the inner periphery of the second region according to thefunction of the distance from the gaze position. The term “baselinesub-sampling pattern” refers to an initial sub-sampling pattern that isused to generate the predetermined sub-sampling pattern. Optionally, asize of the baseline sub-sampling pattern is same as a size of thepredetermined sub-sampling pattern. It will be appreciated that thebaseline sub-sampling pattern of a requisite size could be generated byrepeating an M×N sub-sampling pattern of a smaller size, in at least onegiven direction. The M×N sub-sampling pattern could be, for example, a4×2 sub-sampling pattern, an 8×8 sub-sampling pattern, a 10×10sub-sampling pattern, a 16×12 sub-sampling pattern, a 32×32 sub-samplingpattern, or similar. The number of additional pixels to be included inthe baseline set per unit area varies in the aforesaid manner becausethe pixels near the inner periphery of the second region would beperceived in the at least one image frame with high visual acuity, ascompared to the pixels near the outer periphery of the second region.Therefore, a higher number of additional pixels per unit area arerequired to be included near a portion of the baseline sub-samplingpattern that corresponds to the inner periphery as compared to a portionof the baseline sub-sampling pattern that corresponds to the outerperiphery. The predetermined sub-sampling pattern thus generated enablesin performing spatially variable sub-sampling across the second region,in a manner that mimics how humans focus within their field of view. Inthis regard, the pixels near the inner periphery of the second region(i.e., gaze-proximal pixels) are sampled more than the pixels near theouter periphery of the second region.

Optionally, the baseline sub-sampling pattern indicates locations ofgreen pixels, red pixels and blue pixels (namely, pixels that correspondto a green colour filter, a red colour filter and a blue colour filter,respectively) that are to be read out, wherein the baseline set includesthe green pixels, the red pixels and the blue pixels in a ratio of2:1:1. Moreover, optionally, the predetermined sub-sampling patternindicates locations of green pixels, red pixels and blue pixels of theset that are to be read out from the second region, wherein said setincludes the green pixels, the red pixels and the blue pixels in a ratioof 2:1:1.

Optionally, the image sensor comprises a Bayer colour filter arranged infront of the plurality of pixels of the image sensor. When the pluralityof pixels of the image sensor are arranged in a rectangular 2D grid, theBayer colour filter may have an alternate arrangement of red colourfilters and green colour filters for odd rows of the rectangular 2Dgrid, and an alternate arrangement of green colour filters and bluecolour filters for even rows of the rectangular 2D grid. The Bayercolour filter is well-known in the art. It will be appreciated that whenthe aforesaid baseline sub-sampling pattern is used to generate thepredetermined sub-sampling pattern, a spatially-variable sub-samplingcan be accurately and requisitely performed across the second region. Inan example, when the sub-sampling density is 25 percent near the outerperiphery of the second region, the baseline sub-sampling patternindicates four pixels to be read out from amongst every 16 pixels (of a4×4 grid) of the second region, wherein two out of the four pixelscorrespond to the green colour filters, one out of the four pixelscorresponds to the red colour filters, and one out of the four pixelscorresponds to the blue colour filters. When the sub-sampling density is50 percent near the inner periphery of the second region, the baselinesub-sampling pattern indicates eight pixels to be read out from amongstevery 16 pixels (of the 4×4 grid) of the second region, wherein four outof the eight pixels correspond to the green colour filters, two out ofthe eight pixels correspond to the red colour filters, and two out ofthe eight pixels correspond to the blue colour filters.

Referring to and continuing from the first example, the 8×8 sub-samplingpattern may be repeated 375000 times in an annular manner across bothhorizontal and vertical dimensions to form the baseline sub-samplingpattern, wherein an outer horizontal dimension, an outer verticaldimension, an inner horizontal dimension, and an inner verticaldimension of the baseline sub-sampling pattern may be equal to 5000pixels, 5000 pixels, 1000 pixels, and 1000 pixels, respectively. Thebaseline sub-sampling pattern may have a sampling density equal to 25percent. When the predetermined sub-sampling pattern is generated fromthe baseline sub-sampling pattern, a sub-sampling density near the outerperiphery of the second region may be 25 percent, and a required numberof additional pixels per unit area may be included in the baseline setof the baseline sub-sampling pattern to achieve a sub-sampling densityof 50 percent near the inner periphery of the second region.

Optionally, the processor is configured to:

-   -   identify at least one salient feature in at least one        previously-generated image frame; and    -   determine a given pixel in the second region that represents a        part of the at least one salient feature in the at least one        previously-generated image frame as an additional pixel to be        read out.

The term “salient feature” refers to a feature in a given image framethat is visually alluring (namely, has high saliency). Examples of theat least one salient feature may include, but are not limited to, anedge, a corner, a high-frequency texture detail. Optionally, whenidentifying the at least one salient feature in the at least onepreviously-generated image frame, the processor is configured to employat least one feature-extraction algorithm. Examples of the at least onefeature extraction algorithm include, but are not limited to, anedge-detection algorithm (for example, such as a biased Sobel gradientestimator, a Canny edge detector, Deriche edge detector, and the like),a corner-detection algorithm (for example, such as Harris & Stephenscorner detector, Shi-Tomasi corner detector, Features from AcceleratedSegment Test (FAST) corner detector, and the like), a feature descriptoralgorithm (for example, such as Binary Robust Independent ElementaryFeatures (BRIEF), Gradient Location and Orientation Histogram (GLOH),Histogram of Oriented Gradients (HOG), and the like), and a featuredetector algorithm (for example, such as Scale-Invariant FeatureTransform (SIFT), Oriented FAST and rotated BRIEF (ORB), Speeded UpRobust Features (SURF), and the like).

It will be appreciated that since the at least one salient feature isvisually alluring, the user is more likely to focus on the at least onesalient feature as compared to other features in the at least one imageframe (that is subsequently generated after the at least onepreviously-generated image frame). Therefore, the at least one salientfeature should be represented with high visual quality in the at leastone image frame. For example, the user is more likely to focus on edges,corners, or high-frequency texture details as compared to interiorfeatures, blobs, or low-frequency texture details, since the formertypes of features are more visually alluring as compared to the latter.Therefore, the given pixel is read out to obtain accurate pixel data forthe part of the at least one salient feature so as to obtain a highvisual quality of the at least one salient feature in the at least oneimage frame. Furthermore, when the given pixel is read out,interpolation filtering is performed along the at least one salientfeature, but not across it. Beneficially, in such a case, whendemosaicking is subsequently performed after the interpolationfiltering, undesirable visual artifacts (such as colour bleeding) in theat least one image frame are prevented, which otherwise would have beennoticeable to the user when the user would have viewed the at least onesalient feature in the at least one image frame.

Notably, the processor selectively reads out pixel data from the pixelsof the second region. In particular, the processor reads out the secondpixel data from the pixels of the selected set, instead of reading outpixel data from all pixels of the second region. It will be appreciatedthat such a selective read out of the pixel data facilitates inproviding a high frame rate of image frames. The frame rate is expressedin terms of frames per second (FPS), and may, for example, be 60 FPS, 90FPS, 120 FPS, or higher. This is because a processing time forselectively reading out the pixel data from the pixels of the secondregion and generating the pixel data of the remaining pixels of thesecond region from the second pixel data, is considerably lesser than aprocessing time for reading out the pixel data from each pixel of thesecond region. Therefore, in a given time duration, a higher number ofimage frames could be generated and displayed when the pixel data fromthe pixels of the second region is selectively read out, as compared towhen the pixel data from all pixels of the second region is read out.

Notably, the pixel data of the remaining pixels (i.e., pixels excludedfrom the set of pixels) of the second region is generated using thesecond pixel data. Optionally, the processor is configured to generatethe pixel data of the remaining pixels of the second region byperforming at least one of: interpolation filtering, in-painting. Inthis way, the pixel data of the remaining pixels of the second region isaccurately generated by performing the interpolation filtering and/orthe in-painting on the second pixel data (which may be in the form ofthe RAW image data or the given colour space data), whilst alsoachieving a spatially-variable resolution in the at least one imageframe, and a high frame rate of the image frames. The spatially-variableresolution is achieved by full sampling of the first region andsub-sampling of the second region, as described earlier. The“interpolation filtering” and “in-painting” are specialized processes ofreconstructing damaged, missing, or un-read pixel data of some pixels byusing pixel data read out from other pixels. The interpolation filteringand the in-painting are well-known in the art.

Optionally, the processor is configured to perform the interpolationfiltering by employing at least one interpolation filtering algorithm.Optionally, the at least one interpolation filtering algorithm is atleast one of: a bilinear interpolation algorithm, an edge-directedweighted-sum interpolation algorithm, a weighted sum interpolationalgorithm, a local colour ratio (LCR) algorithm, a median-basedinterpolation algorithm, an average-based interpolation algorithm, alinear interpolation filtering algorithm, a cubic interpolationfiltering algorithm, a four-nearest-neighbours interpolation filteringalgorithm, a natural-neighbour interpolation filtering algorithm, asteering kernel regression interpolation filtering algorithm. Someinterpolation filtering algorithms that are used in the demosaickingprocess may also be used to perform the interpolation filtering. The LCRalgorithm may be used for generating a red colour value and/or a bluecolour value in the pixel data of the remaining pixels of the secondregion. It will be appreciated that the edge-directed weighted-suminterpolation algorithm takes into account edge behaviours around theremaining pixels of the second region whose pixel data is to begenerated by employing said algorithm. One such algorithm is described,for example, in “Digital Camera Zooming Based on Unified CFA ImageProcessing Steps” by R. Lukac, K. Martin, and K. N. Plataniotis,published in IEEE Transactions on Consumer Electronics, Vol. 50, No. 1,pp. 15-24, February 2004, which has been incorporated herein byreference.

Optionally, the processor is configured to perform the in-painting byemploying at least one in-painting algorithm. Optionally, the at leastone in-painting algorithm is at least one of: a Rapid FrequencySelective Reconstruction (FSR) algorithm, a Fast Marching Method (FMM)algorithm, a Navier Stokes (NS)-based in-painting algorithm, a coherencetransport-based in-painting algorithm, an exemplar-based in-paintingalgorithm, a Criminisi's algorithm, a group-based sparse representation(GSR) algorithm, a compression-oriented edge-based in-paintingalgorithm, an annihilating filter-based low-rank Hankel matrix approach(ALOHA) algorithm, an image melding using patch-based synthesisalgorithm.

Optionally, the processor is configured to perform at least oneprocessing operation prior to or after generating the pixel data ofremaining pixels of the second region from the second pixel data. Inthis regard, some processing operations may be performed prior to thegeneration of the pixel data of the remaining pixels, while otherprocessing operations may be performed after the generation of the pixeldata of the remaining pixels. Optionally, the at least one processingoperation is at least one of: black level correction, defective pixelcorrection (DPC), Bayer domain denoising, lens shading correction,scaling, automatic white balance gain adjustment, demosaicking,automatic white balance static parameters adjustment, colour conversionmatrix interpolation, autofocus, auto exposure, gamma correction, colourspace conversion, luma and chroma denoising, sharpening and edgeenhancement, contrast adjustment, shot noise correction, chromaticaberration correction, reprojection.

In an exemplary implementation, the interpolation filtering is performedon the second pixel data prior to the demosaicking. In another exemplaryimplementation, the demosaicking and the interpolation filtering arecombined as a single operation. In yet another exemplary implementation,the in-painting is performed on the second pixel data separately foreach colour channel of a given colour space (such as for a red colourchannel, a green colour channel, and a blue colour channel of the RGBcolour space) after the demosaicking.

Typically, processing of the first pixel data, the second pixel data andthe generated pixel data is performed, by the processor, in differentsteps. Optionally, when processing the first pixel data, the secondpixel data and the generated pixel data to generate at least one imageframe, the processor is configured to employ at least one imageprocessing algorithm. Examples of the at least one image processingalgorithm include, but are not limited to, an image denoising algorithm,an image sharpening algorithm, a colour conversion algorithm, and anauto white balancing algorithm. Different pixel data corresponding todifferent parts of the at least one image frame are processed by theprocessor in a manner that processor overburdening, delays, andexcessive power consumption do not occur during said processing. In thisregard, conservation and utilization of processing resources of theprocessor is optimized. An image quality of the at least one image frameso generated emulates image viewing quality and characteristics of humanvisual system. In particular, the at least one image frame has thespatially variable resolution, wherein a first portion of the at leastone image frame corresponding to the first region has a first resolutionthat is higher than a second resolution of a second portion of the atleast one image frame corresponding to the second region. Optionally,the first resolution is greater than or equal to 30 pixels per degree(PPD), whereas the second resolution is greater than or equal to 10 PPD.As an example, the first resolution may be 60 PPD, whereas the secondresolution varies spatially within a range of 15 PPD (at an outerperiphery of the second portion) to 30 PPD (at an inner periphery of thesecond portion).

Notably, the at least one image frame is displayed via the light sourceof the display apparatus. Herein, the term “light source” refers to anelement from which light emanates. Optionally, a given light source isimplemented as a display. In this regard, a given image frame isdisplayed at the display. Examples of such a display include, but arenot limited to, a Liquid Crystal Display (LCD), a Light-Emitting Diode(LED)-based display, an Organic LED (OLED)-based display, a microOLED-based display, an Active Matrix OLED (AMOLED)-based display, and aLiquid Crystal on Silicon (LCoS)-based display. Alternatively,optionally, a given light source is implemented as a projector. In thisregard, a given image frame is projected onto a projection screen ordirectly onto a retina of the user's eyes. Examples of such a projectorinclude, but are not limited to, an LCD-based projector, an LED-basedprojector, an OLED-based projector, an LCoS-based projector, a DigitalLight Processing (DLP)-based projector, and a laser projector.Optionally, the given light source is a single-resolution light sourceor a multi-resolution light source.

Optionally, the processor is configured to:

-   -   obtain information indicative of a head pose of the user;    -   detect whether or not a change in the head pose of the user is        greater than a predefined angle within a predefined time period;    -   when it is detected that the change in the head pose is greater        than the predefined angle within the predefined time period,        employ a first sub-sampling pattern whose maximum sub-sampling        density is equal to a first sub-sampling density as the        predetermined sub-sampling pattern; and    -   when it is detected that the change in the head pose is not        greater than the predefined angle within the predefined time        period, employ a second sub-sampling pattern whose maximum        sub-sampling density is equal to a second sub-sampling density        as the predetermined sub-sampling pattern, wherein the second        sub-sampling density is higher than the first sub-sampling        density.

Optionally, the processor obtains the information indicative of the headpose of the user from a pose-tracking means of the display apparatus.The term “pose-tracking means” refers to specialized equipment that isemployed to detect and/or follow a head pose of the user within thereal-world environment, when the user wears the display apparatus onhis/her head. The term “pose” encompasses both position and orientation.In practice, the pose-tracking means is actually employed to track apose of the HMD; the head pose of the user corresponds to the pose ofthe HMD as the HMD is worn by the user on his/her head. Pursuant toembodiments of the present disclosure, the pose-tracking means isimplemented as a true six Degrees of Freedom (6DoF) tracking system. Inother words, the pose-tracking means tracks the pose of the user's headwithin a 3D space of the real-world environment, wherein pose-trackingdata constitutes the information indicative of the head pose of theuser. In particular, said pose-tracking means is configured to tracktranslational movements (namely, surge, heave and sway movements) androtational movements (namely, roll, pitch and yaw movements) of theuser's head within the 3D space.

The pose-tracking means could be implemented as an internal component ofthe HMD, as a tracking system external to the HMD, or as a combinationthereof. The pose-tracking means could be implemented as at least oneof: an optics-based tracking system (which utilizes, for example,infrared beacons and detectors, infrared cameras, visible-light cameras,detectable objects and detectors, and the like), an acoustics-basedtracking system, a radio-based tracking system, a magnetism-basedtracking system, an accelerometer, a gyroscope, an Inertial MeasurementUnit (IMU), a Timing and Inertial Measurement Unit (TIMU). As anexample, a detectable object may be an active infra-red (IR) LED, avisible LED, a laser illuminator, a Quick Response (QR) code, an ArUcomarker, an anchor marker, a Radio Frequency Identification (RFID)marker, and the like. A detector may be implemented as at least one of:an IR camera, an IR transceiver, a visible light camera, an RFID reader.Optionally, a given processor employs at least one data processingalgorithm to process the pose-tracking data, to determine a head pose ofthe user. The pose-tracking data may be in form of images, IMU/TIMUvalues, motion sensor data values, magnetic field strength values, orsimilar. Correspondingly, requisite data processing algorithm(s) is/areemployed to process the pose-tracking data, to determine the head poseof the user. Examples of the at least one data processing algorithminclude a feature detection algorithm, an environment mapping algorithm,a data extrapolation algorithm, and the like.

Optionally, the predefined angle within the predefined time period liesin a range of 10 degrees within the predefined time period to 150degrees within the predefined time period. Optionally, the predefinedtime period lies in a range of 50 milliseconds to 700 milliseconds. Asan example, the predefined angle within the predefined time period maybe 30 degrees within 500 milliseconds, 20 degrees within 150milliseconds, or similar.

In one case, when it is detected that the change in the head pose isgreater than the predefined angle within the predefined time period, theuser's head is moving rapidly, it may be determined that the user wouldperceive a low visual quality in the non-gaze-contingent pixels of theat least one image frame. Therefore, pixel data from the pixels lying inthe second region is selectively read out according to the firstsub-sampling pattern. In such a case, the first sub-sampling densityenables in achieving the (required) low visual quality in thenon-gaze-contingent pixels of the at least one image frame.

In another case, when it is detected that the change in the head pose isnot greater than the predefined angle within the predefined time period,the user's head is moving slowly as the user may be concentrating on acertain region in the visual scene. In such a case, it may be determinedthat the user would be able to perceive considerable visual quality inthe non-gaze-contingent pixels of the at least one image frame.Therefore, pixel data from the pixels lying in the second region isselectively read out according to the second sub-sampling pattern whosesecond sub-sampling density is higher than the first sub-samplingdensity. As a result, a relatively higher visual quality is achieved inthe non-gaze-contingent pixels of the at least one image frame ascompared to when the first sub-sampling density is employed.

Optionally, when it is detected that the change in the head pose isgreater than the predefined angle within the predefined time period, theprocessor is configured to:

-   -   select a set of pixels that are to be read out from the first        region based on a third sub-sampling pattern having a third        sub-sampling density, wherein the third sub-sampling pattern        indicates locations of the pixels of the set, the third        sub-sampling density being higher than the second sub-sampling        density and the first sub-sampling density;    -   read out only a portion of the first pixel data from the pixels        of the selected set, instead of an entirety of the first pixel        data; and    -   generate, from said portion of the first pixel data, a remaining        portion of the first pixel data.

In this regard, the pixels lying in the first region are sub-sampled(according to the third sub-sampling pattern). The sub-sampling wouldreduce visual quality of the gaze-contingent pixels of the at least oneimage frame, but such a reduction in the visual quality would not beperceivable by the user as the user's head is moving very fast.Therefore, the user's viewing experience would not be compromised, butprocessing resource savings would be achieved. It will be appreciatedthat the subsampling in the first region enables in optimizingutilization of processing resources and reducing processing time of theprocessor. This is because the processing resources and the processingtime required in reading out the portion of the first pixel data areconsiderably less as compared to those that are required when theentirety of the first pixel data is read. Optionally, the processor isconfigured to generate the remaining portion of the first pixel datafrom the portion of the first pixel data, by performing at least one of:interpolation filtering, in-painting.

The present disclosure also relates to the display apparatus asdescribed above. Various embodiments and variants disclosed above applymutatis mutandis to the display apparatus.

Optionally, in the display apparatus, the predetermined sub-samplingpattern is a non-regular pattern.

Optionally, in the display apparatus, the at least one processor isconfigured to change the predetermined sub-sampling pattern from oneimage frame to another image frame.

Optionally, in the display apparatus, a sub-sampling density of thepredetermined sub-sampling pattern varies across the second region as afunction of a distance from the gaze position.

Optionally, in the display apparatus, the at least one processor isconfigured to generate the predetermined sub-sampling pattern from abaseline sub-sampling pattern having a same sub-sampling density acrossthe second region and indicating locations of pixels of a baseline set,by including additional pixels in the baseline set and indicatinglocations of the additional pixels, wherein a number of additionalpixels to be included in the baseline set per unit area increases ongoing from an outer periphery of the second region towards an innerperiphery of the second region according to the function of the distancefrom the gaze position.

Optionally, in the display apparatus, the baseline sub-sampling patternindicates locations of green pixels, red pixels and blue pixels that areto be read out, wherein the baseline set includes the green pixels, thered pixels and the blue pixels in a ratio of 2:1:1.

Optionally, in the display apparatus, the at least one processor isconfigured to:

-   -   identify at least one salient feature in at least one        previously-generated image frame; and    -   determine a given pixel in the second region that represents a        part of the at least one salient feature in the at least one        previously-generated image frame as an additional pixel to be        read out.

Optionally, the display apparatus further comprises pose-tracking means,wherein the at least one processor is configured to:

-   -   process pose-tracking data, collected by the pose-tracking        means, to determine a head pose of the user;    -   detect whether or not a change in the head pose of the user is        greater than a predefined angle within a predefined time period;    -   when it is detected that the change in the head pose is greater        than the predefined angle within the predefined time period,        employ a first sub-sampling pattern whose maximum sub-sampling        density is equal to a first sub-sampling density as the        predetermined sub-sampling pattern; and    -   when it is detected that the change in the head pose is not        greater than the predefined angle within the predefined time        period, employ a second sub-sampling pattern whose maximum        sub-sampling density is equal to a second sub-sampling density        as the predetermined sub-sampling pattern, wherein the second        sub-sampling density is higher than the first sub-sampling        density.

Optionally, in the display apparatus, when it is detected that thechange in the head pose is greater than the predefined angle within thepredefined time period, the at least one processor is configured to:

-   -   select a set of pixels that are to be read out from the first        region based on a third sub-sampling pattern having a third        sub-sampling density, wherein the third sub-sampling pattern        indicates locations of the pixels of the set, the third        sub-sampling density being higher than the second sub-sampling        density and the first sub-sampling density;    -   read out only a portion of the first pixel data from the pixels        of the selected set, instead of an entirety of the first pixel        data; and    -   generate, from said portion of the first pixel data, a remaining        portion of the first pixel data.

Optionally, in the display apparatus, the at least one processor isconfigured to generate the pixel data of the remaining pixels of thesecond region by performing at least one of: interpolation filtering,in-painting.

The present disclosure also relates to the method as described above.Various embodiments and variants disclosed above apply mutatis mutandisto the method.

Optionally, in the method, the predetermined sub-sampling pattern is anon-regular pattern.

Optionally, the method further comprises changing the predeterminedsub-sampling pattern from one image frame to another image frame.

Optionally, in the method, a sub-sampling density of the predeterminedsub-sampling pattern varies across the second region as a function of adistance from the gaze position.

Optionally, the method further comprises generating the predeterminedsub-sampling pattern from a baseline sub-sampling pattern having a samesub-sampling density across the second region and indicating locationsof pixels of a baseline set, by including additional pixels in thebaseline set and indicating locations of the additional pixels, whereina number of additional pixels to be included in the baseline set perunit area increases on going from an outer periphery of the secondregion towards an inner periphery of the second region according to thefunction of the distance from the gaze position.

Optionally, in the method, the baseline sub-sampling pattern indicateslocations of green pixels, red pixels and blue pixels that are to beread out, wherein the baseline set includes the green pixels, the redpixels and the blue pixels in a ratio of 2:1:1.

Optionally, the method further comprises:

-   -   identifying at least one salient feature in at least one        previously-generated image frame; and    -   determining a given pixel in the second region that represents a        part of the at least one salient feature in the at least one        previously-generated image frame as an additional pixel to be        read out.

Optionally, the method further comprises:

-   -   obtaining information indicative of a head pose of the user;    -   detecting whether or not a change in the head pose of the user        is greater than a predefined angle within a predefined time        period;    -   when it is detected that the change in the head pose is greater        than the predefined angle within the predefined time period,        employing a first sub-sampling pattern whose maximum        sub-sampling density is equal to a first sub-sampling density as        the predetermined sub-sampling pattern; and    -   when it is detected that the change in the head pose is not        greater than the predefined angle within the predefined time        period, employing a second sub-sampling pattern whose maximum        sub-sampling density is equal to a second sub-sampling density        as the predetermined sub-sampling pattern, wherein the second        sub-sampling density is higher than the first sub-sampling        density.

Optionally, when it is detected that the change in the head pose isgreater than the predefined angle within the predefined time period, themethod further comprises:

-   -   selecting a set of pixels that are to be read out from the first        region based on a third sub-sampling pattern having a third        sub-sampling density, wherein the third sub-sampling pattern        indicates locations of the pixels of the set, the third        sub-sampling density being higher than the second sub-sampling        density and the first sub-sampling density;    -   reading out only a portion of the first pixel data from the        pixels of the selected set, instead of an entirety of the first        pixel data; and    -   generating, from said portion of the first pixel data, a        remaining portion of the first pixel data.

Optionally, the method further comprises generating the pixel data ofthe remaining pixels of the second region by performing at least one of:interpolation filtering, in-painting.

DETAILED DESCRIPTION OF THE DRAWINGS

Referring to FIG. 1 , illustrated is a block diagram of an architectureof an imaging system 100, in accordance with an embodiment of thepresent disclosure. The imaging system 100 comprises an image sensor 102and a processor 104. The image sensor 102 comprises a plurality ofpixels (not shown) arranged on a photo-sensitive surface thereof. Theprocessor 104 is coupled to the image sensor 102.

Referring to FIGS. 2A and 2B, illustrated are block diagrams ofarchitectures of a display apparatus 200, in accordance with differentembodiments of the present disclosure. The display apparatus 200comprises a gaze-tracking means 202, a light source per eye (depicted asa light source 204 corresponding to a left eye and a light source 206corresponding to a right eye), an image sensor per eye (depicted as animage sensor 208 corresponding to the left eye and an image sensor 210corresponding to the right eye), and at least one processor (depicted asa processor 212). The processor 212 is coupled to the gaze-trackingmeans 202, the light sources 204 and 206, and the image sensors 208 and210. In FIG. 2B, the display apparatus 200 further comprisespose-tracking means 214. The processor 212 is also coupled to thepose-tracking means 214.

Referring to FIGS. 3A and 3B, FIG. 3A illustrates different regions on aphoto-sensitive surface 300 of an image sensor, while FIG. 3Billustrates how pixels arranged on the photo-sensitive surface 300 areread out, in accordance with an embodiment of the present disclosure. InFIG. 3A, the photo-sensitive surface 300 is shown to comprise 576 pixelsarranged in a 24×24 grid, for the sake of simplicity and clarity only.It will be appreciated that a photo-sensitive surface of a typical imagesensor has millions of pixels. The photo-sensitive surface 300 has afirst region 302 and a second region 304 that are determined based on agaze position (for example, at a centre of the photo-sensitive surface300). The first region 302 includes and surrounds the gaze position andis shown as an 8×8 grid of 64 pixels. The second region 304 surroundsthe first region 302 and is shown to include 512 pixels. The secondregion 304 includes a first sub-region 304A and a second sub-region304B, wherein the first sub-region 304A lies between the secondsub-region 304B and the first region 302. The first sub-region 304A isshown to include 192 pixels and the second sub-region 304B is shown toinclude 320 pixels.

In FIG. 3B, white pixels illustrate pixels that are read out whereasblackened pixels illustrate pixels that are not read out. Notably, eachpixel of the first region 302 is read out to constitute first pixeldata, whereas only a set of pixels are read out from the second region304 to constitute second pixel data. The set of pixels are selectedbased on a predetermined sub-sampling pattern. A sampling density acrossthe first region 302 is 100 percent (i.e., all 64 pixels in the firstregion 302 are read out). A sub-sampling density of the predeterminedsub-sampling pattern varies across the second region 304 as a functionof a distance from the gaze position. For example, a sub-samplingdensity in the first sub-region 304A is 50 percent (i.e., half of the192 pixels in the first sub-region 304A are read out), while asub-sampling density in the second sub-region 304B is 25 percent (i.e.,a quarter of the 320 pixels in the second sub-region 304B are read out).

FIGS. 3A and 3B are merely examples, which should not unduly limit thescope of the claims herein. A person skilled in the art will recognizemany variations, alternatives, and modifications of embodiments of thepresent disclosure. In an example, photo-sensitive surface 300 maycomprise 25 megapixels arranged in a 5000×5000 grid.

Referring to FIG. 4 , illustrated is a manner in which a predeterminedsub-sampling pattern changes across eight image frames, in accordancewith an embodiment of the present disclosure. The predeterminedsub-sampling pattern is a non-regular pattern. For sake of simplicity,only a portion of the predetermined sub-sampling pattern correspondingto a 4×4 grid of pixels of the second region is depicted in FIG. 4 .Various forms of the portion of the predetermined sub-sampling patterncorresponding to the eight image frames are represented as P1, P2, P3,P4, P5, P6, P7, and P8. A sub-sampling density of the portion of thepredetermined sub-sampling pattern is shown to be 25 percent (i.e., onlypixels that are crossed out as ‘X’ are read out while other pixels arenot read out).

Referring to FIG. 5 , illustrated is a sub-sampling pattern 500 that isused to generate a baseline sub-sampling pattern (not shown), inaccordance with an embodiment of the present disclosure. Thesub-sampling pattern 500 is an 8×8 sub-sampling pattern having asub-sampling density of 25 percent (i.e., only 16 pixels depicted aswhite pixels are read out while remaining 48 pixels depicted asblackened pixels are not read out). The baseline sub-sampling pattern ofa requisite size could be generated by repeating the sub-samplingpattern 500, in at least one given direction. Notably, the baselinesub-sampling pattern is used to generate a predetermined sub-samplingpattern.

Referring to FIGS. 6A, 6B, 6C, and 6D, illustrated is a manner in whichpixel data of remaining pixels of a portion 600 of a second region on aphoto-sensitive surface (not shown) is generated, in accordance with anembodiment of the present disclosure. The portion 600 of the secondregion is shown to include 64 pixels arranged in an 8×8 grid, for thesake of simplicity. In FIGS. 6A-6D, pixel data of the remaining pixelsof the portion 600 is shown, for example, to be generated by performinginterpolation filtering.

In FIG. 6A, pixels that are read out from the portion 600 of the secondregion are shown as 16 pixels that are crossed out as solid ‘X’s. Adiamond pattern 602 is employed for generating pixel data, wherein pixeldata of pixels at corners of the diamond pattern 602 is used to generatepixel data of a pixel (labelled as ‘+’) lying in a centre of the diamondpattern 602. The diamond pattern 602 is moved across the portion 600, togenerate pixel data of at least one pixel in the portion 600. In FIG.6B, there are shown 8 pixels (crossed out as dotted ‘X’s) for whichpixel data is generated using the diamond pattern 602. Next, in FIG. 6C,pixels for which pixel data is available are shown as 24 pixels that arecrossed out as solid ‘X’s. A square pattern 604 is employed forgenerating pixel data, wherein pixel data of pixels at corners of thesquare pattern 604 is used to generate pixel data of a pixel (labelledas ‘+’) lying in a centre of the square pattern 604. The square pattern604 is moved across the portion 600, to generate pixel data of at leastone pixel in the portion 600. In FIG. 6D, there are shown 16 pixels(crossed out as dotted ‘X’s) for which pixel data is generated using thesquare pattern 604. It will be appreciated that apart from using thediamond pattern and the square pattern as described hereinabove, when aremaining pixel (of the portion 600) lies diagonally in between twopixels that are read out from the portion 600, pixel data of theremaining pixel could be generated by averaging pixel data of the twopixels that are read out from the portion 600, or by performing thelinear interpolation filtering on the pixel data of said two pixels.

It may be understood by a person skilled in the art that the FIGS. 6A,6B, 6C, and 6D are merely examples for sake of clarity, which should notunduly limit the scope of the claims herein. The person skilled in theart will recognize many variations, alternatives, and modifications ofembodiments of the present disclosure.

Referring to FIGS. 7A and 7B, illustrated is a processing pipelineindicating where generation of pixel data of remaining pixels of asecond region is performed, in accordance with different embodiments ofthe present disclosure. The processing pipeline is implemented by aprocessor 702 that is coupled to an image sensor 704, wherein the imagesensor 704 comprises a plurality of pixels arranged on a photo-sensitivesurface thereof. The processor 702 is implemented, for example, as animage signal processor. The processing pipeline includes severalprocessing operations (depicted, for example, as processing operations706, 708, 710, and 712). For example, the processing operation 706 isblack level correction and shading correction, the processing operation708 is the generation of pixel data of remaining pixels of the secondregion, the processing operation 710 is demosaicking, and the processingoperation 712 is at least one image enhancement operation. Some of theseveral processing operations are performed in a RAW domain 714, whileothers are performed in a colour domain 716. The colour domain 716 may,for example, be a Red Green Blue (RGB) domain. In FIG. 7A, thegeneration of pixel data of remaining pixels of the second region (i.e.,the processing operation 708) is performed in the RAW domain 714. InFIG. 7B, the generation of pixel data of remaining pixels of the secondregion (i.e., the processing operation 708) is performed in the colourdomain 716.

It may be understood by a person skilled in the art that the FIGS. 7Aand 7B are merely examples for sake of clarity, which should not undulylimit the scope of the claims herein. The person skilled in the art willrecognize many variations, alternatives, and modifications ofembodiments of the present disclosure. For example, the processingpipeline may include several other processing operations, for example,defective pixel correction (DPC), Bayer domain denoising, lens shadingcorrection, scaling, automatic white balance gain adjustment, and thelike.

Referring to FIGS. 8A and 8B, illustrated are steps of a method, inaccordance with an embodiment of the present disclosure. At step 802,information indicative of a gaze direction of a user's eye is obtained.At step 804, a gaze position on a photo-sensitive surface of an imagesensor is identified based on the gaze direction of the user's eye. Atstep 806, a first region and a second region are determined on thephoto-sensitive surface of the image sensor based on the gaze position,wherein the first region includes and surrounds the gaze position whilethe second region surrounds the first region. At step 808, first pixeldata is read out from each pixel of the first region. At step 810, a setof pixels that are to be read out from the second region are selectedbased on a predetermined sub-sampling pattern, wherein the predeterminedsub-sampling pattern indicates locations of the pixels of the set. Atstep 812, second pixel data is read out from the pixels of the selectedset. At step 814, pixel data of remaining pixels of the second region isgenerated from the second pixel data. At step 816, the first pixel data,the second pixel data and the generated pixel data are processed togenerate at least one image frame.

The steps 802, 804, 806, 808, 810, 812, 814, and 816 are onlyillustrative and other alternatives can also be provided where one ormore steps are added, one or more steps are removed, or one or moresteps are provided in a different sequence without departing from thescope of the claims herein.

Modifications to embodiments of the present disclosure described in theforegoing are possible without departing from the scope of the presentdisclosure as defined by the accompanying claims. Expressions such as“including”, “comprising”, “incorporating”, “have”, “is” used todescribe and claim the present disclosure are intended to be construedin a non-exclusive manner, namely allowing for items, components orelements not explicitly described also to be present. Reference to thesingular is also to be construed to relate to the plural.

The invention claimed is:
 1. An imaging system comprising: an imagesensor comprising a plurality of pixels arranged on a photo-sensitivesurface thereof; and a processor configured to: obtain informationindicative of a gaze direction of a user's eye; identify a gaze positionon the photo-sensitive surface of the image sensor, based on the gazedirection of the user's eye; determine a first region and a secondregion on the photo-sensitive surface of the image sensor based on thegaze position, wherein the first region includes and surrounds the gazeposition, while the second region surrounds the first region; read outfirst pixel data from each pixel of the first region; select a set ofpixels that are to be read out from the second region based on apredetermined spatially irregular sub-sampling pattern, wherein thepredetermined sub-sampling pattern indicates locations of the pixels ofthe set; read out second pixel data from the pixels of the selected set;generate, from the second pixel data, pixel data of remaining pixels ofthe second region; and process the first pixel data, the second pixeldata and the generated pixel data to generate at least one image frame.2. The imaging system of claim 1, wherein the predetermined sub-samplingpattern is a non-regular pattern.
 3. The imaging system of claim 1,wherein the processor is configured to change the predeterminedsub-sampling pattern from one image frame to another image frame.
 4. Theimaging system of claim 1, wherein a sub-sampling density of thepredetermined sub-sampling pattern varies across the second region as afunction of a distance from the gaze position.
 5. The imaging system ofclaim 4, wherein the sub-sampling density is at least 25 percent.
 6. Theimaging system of claim 4, wherein the processor is configured togenerate the predetermined sub-sampling pattern from a baselinesub-sampling pattern having a same sub-sampling density across thesecond region and indicating locations of pixels of a baseline set, byincluding additional pixels in the baseline set and indicating locationsof the additional pixels, wherein a number of additional pixels to beincluded in the baseline set per unit area increases on going from anouter periphery of the second region towards an inner periphery of thesecond region according to the function of the distance from the gazeposition.
 7. The imaging system of claim 6, wherein the baselinesub-sampling pattern indicates locations of green pixels, red pixels andblue pixels that are to be read out, wherein the baseline set includesthe green pixels, the red pixels and the blue pixels in a ratio of2:1:1.
 8. The imaging system of claim 6, wherein the processor isconfigured to: identify at least one salient feature in at least onepreviously-generated image frame; and determine a given pixel in thesecond region that represents a part of the at least one salient featurein the at least one previously-generated image frame as an additionalpixel to be read out.
 9. The imaging system of claim 1, wherein theprocessor is configured to: obtain information indicative of a head poseof the user; detect whether or not a change in the head pose of the useris greater than a predefined angle within a predefined time period; whenit is detected that the change in the head pose is greater than thepredefined angle within the predefined time period, employ a firstsub-sampling pattern whose maximum sub-sampling density is equal to afirst sub-sampling density as the predetermined sub-sampling pattern;and when it is detected that the change in the head pose is not greaterthan the predefined angle within the predefined time period, employ asecond sub-sampling pattern whose maximum sub-sampling density is equalto a second sub-sampling density as the predetermined sub-samplingpattern, wherein the second sub-sampling density is higher than thefirst sub-sampling density.
 10. The imaging system of claim 9, whereinwhen it is detected that the change in the head pose is greater than thepredefined angle within the predefined time period, the processor isconfigured to: select a set of pixels that are to be read out from thefirst region based on a third sub-sampling pattern having a thirdsub-sampling density, wherein the third sub-sampling pattern indicateslocations of the pixels of the set, the third sub-sampling density beinghigher than the second sub-sampling density and the first sub-samplingdensity; read out only a portion of the first pixel data from the pixelsof the selected set, instead of an entirety of the first pixel data; andgenerate, from said portion of the first pixel data, a remaining portionof the first pixel data.
 11. The imaging system of claim 1, wherein theprocessor is configured to generate the pixel data of the remainingpixels of the second region by performing at least one of: interpolationfiltering, in-painting.
 12. A display apparatus comprising:gaze-tracking means; a light source per eye; an image sensor per eyecomprising a plurality of pixels arranged on a photo-sensitive surfacethereof; and at least one processor configured to: process gaze-trackingdata, collected by the gaze-tracking means, to determine a gazedirection of a user's eye; identify a gaze position on thephoto-sensitive surface of the image sensor, based on the gaze directionof the user's eye; determine a first region and a second region on thephoto-sensitive surface of the image sensor based on the gaze position,wherein the first region includes and surrounds the gaze position, whilethe second region surrounds the first region; read out first pixel datafrom each pixel of the first region; select a set of pixels that are tobe read out from the second region based on a predetermined spatiallyirregular sub-sampling pattern, wherein the predetermined sub-samplingpattern indicates locations of the pixels of the set; read out secondpixel data from the pixels of the selected set; generate, from thesecond pixel data, pixel data of remaining pixels of the second region;process the first pixel data, the second pixel data and the generatedpixel data to generate at least one image frame; and display the atleast one image frame via the light source.
 13. The display apparatus ofclaim 12, wherein the predetermined sub-sampling pattern is anon-regular pattern.
 14. The display apparatus of claim 12, wherein theat least one processor is configured to change the predeterminedsub-sampling pattern from one image frame to another image frame. 15.The display apparatus of claim 12, wherein a sub-sampling density of thepredetermined sub-sampling pattern varies across the second region as afunction of a distance from the gaze position.
 16. The display apparatusof claim 15, wherein the at least one processor is configured togenerate the predetermined sub-sampling pattern from a baselinesub-sampling pattern having a same sub-sampling density across thesecond region and indicating locations of pixels of a baseline set, byincluding additional pixels in the baseline set and indicating locationsof the additional pixels, wherein a number of additional pixels to beincluded in the baseline set per unit area increases on going from anouter periphery of the second region towards an inner periphery of thesecond region according to the function of the distance from the gazeposition.
 17. The display apparatus of claim 16, wherein the baselinesub-sampling pattern indicates locations of green pixels, red pixels andblue pixels that are to be read out, wherein the baseline set includesthe green pixels, the red pixels and the blue pixels in a ratio of2:1:1.
 18. The display apparatus of claim 16, wherein the at least oneprocessor is configured to: identify at least one salient feature in atleast one previously-generated image frame; and determine a given pixelin the second region that represents a part of the at least one salientfeature in the at least one previously-generated image frame as anadditional pixel to be read out.
 19. The display apparatus of claim 12,further comprising pose-tracking means, wherein the at least oneprocessor is configured to: process pose-tracking data, collected by thepose-tracking means, to determine a head pose of the user; detectwhether or not a change in the head pose of the user is greater than apredefined angle within a predefined time period; when it is detectedthat the change in the head pose is greater than the predefined anglewithin the predefined time period, employ a first sub-sampling patternwhose maximum sub-sampling density is equal to a first sub-samplingdensity as the predetermined sub-sampling pattern; and when it isdetected that the change in the head pose is not greater than thepredefined angle within the predefined time period, employ a secondsub-sampling pattern whose maximum sub-sampling density is equal to asecond sub-sampling density as the predetermined sub-sampling pattern,wherein the second sub-sampling density is higher than the firstsub-sampling density.
 20. The display apparatus of claim 19, whereinwhen it is detected that the change in the head pose is greater than thepredefined angle within the predefined time period, the at least oneprocessor is configured to: select a set of pixels that are to be readout from the first region based on a third sub-sampling pattern having athird sub-sampling density, wherein the third sub-sampling patternindicates locations of the pixels of the set, the third sub-samplingdensity being higher than the second sub-sampling density and the firstsub-sampling density; read out only a portion of the first pixel datafrom the pixels of the selected set, instead of an entirety of the firstpixel data; and generate, from said portion of the first pixel data, aremaining portion of the first pixel data.
 21. The display apparatus ofclaim 12, wherein the at least one processor is configured to generatethe pixel data of the remaining pixels of the second region byperforming at least one of: interpolation filtering, in-painting.
 22. Amethod comprising: obtaining information indicative of a gaze directionof a user's eye; identifying a gaze position on a photo-sensitivesurface of an image sensor, based on the gaze direction of the user'seye; determining a first region and a second region on thephoto-sensitive surface of the image sensor based on the gaze position,wherein the first region includes and surrounds the gaze position, whilethe second region surrounds the first region; reading out first pixeldata from each pixel of the first region; selecting a set of pixels thatare to be read out from the second region based on a predeterminedspatially irregular sub-sampling pattern, wherein the predeterminedsub-sampling pattern indicates locations of the pixels of the set;reading out second pixel data from the pixels of the selected set;generating, from the second pixel data, pixel data of remaining pixelsof the second region; and processing the first pixel data, the secondpixel data and the generated pixel data to generate at least one imageframe.
 23. The method of claim 22, wherein the predeterminedsub-sampling pattern is a non-regular pattern.
 24. The method of claim22, further comprising changing the predetermined sub-sampling patternfrom one image frame to another image frame.
 25. The method of claim 22,wherein a sub-sampling density of the predetermined sub-sampling patternvaries across the second region as a function of a distance from thegaze position.
 26. The method of claim 25, further comprising generatingthe predetermined sub-sampling pattern from a baseline sub-samplingpattern having a same sub-sampling density across the second region andindicating locations of pixels of a baseline set, by includingadditional pixels in the baseline set and indicating locations of theadditional pixels, wherein a number of additional pixels to be includedin the baseline set per unit area increases on going from an outerperiphery of the second region towards an inner periphery of the secondregion according to the function of the distance from the gaze position.27. The method of claim 26, wherein the baseline sub-sampling patternindicates locations of green pixels, red pixels and blue pixels that areto be read out, wherein the baseline set includes the green pixels, thered pixels and the blue pixels in a ratio of 2:1:1.
 28. The method ofclaim 26, further comprising: identifying at least one salient featurein at least one previously-generated image frame; and determining agiven pixel in the second region that represents a part of the at leastone salient feature in the at least one previously-generated image frameas an additional pixel to be read out.
 29. The method of claim 22,further comprising: obtaining information indicative of a head pose ofthe user; detecting whether or not a change in the head pose of the useris greater than a predefined angle within a predefined time period; whenit is detected that the change in the head pose is greater than thepredefined angle within the predefined time period, employing a firstsub-sampling pattern whose maximum sub-sampling density is equal to afirst sub-sampling density as the predetermined sub-sampling pattern;and when it is detected that the change in the head pose is not greaterthan the predefined angle within the predefined time period, employing asecond sub-sampling pattern whose maximum sub-sampling density is equalto a second sub-sampling density as the predetermined sub-samplingpattern, wherein the second sub-sampling density is higher than thefirst sub-sampling density.
 30. The method of claim 29, wherein when itis detected that the change in the head pose is greater than thepredefined angle within the predefined time period, the method furthercomprises: selecting a set of pixels that are to be read out from thefirst region based on a third sub-sampling pattern having a thirdsub-sampling density, wherein the third sub-sampling pattern indicateslocations of the pixels of the set, the third sub-sampling density beinghigher than the second sub-sampling density and the first sub-samplingdensity; reading out only a portion of the first pixel data from thepixels of the selected set, instead of an entirety of the first pixeldata; and generating, from said portion of the first pixel data, aremaining portion of the first pixel data.
 31. The method of claim 22,further comprising generating the pixel data of the remaining pixels ofthe second region by performing at least one of: interpolationfiltering, in-painting.